Skip to content

Conversation

codeflash-ai[bot]
Copy link

@codeflash-ai codeflash-ai bot commented Oct 23, 2025

📄 43% (0.43x) speedup for get_template_variables in guardrails/utils/templating_utils.py

⏱️ Runtime : 3.07 milliseconds 2.14 milliseconds (best of 56 runs)

📝 Explanation and details

The optimization replaces the expensive safe_substitute() operation with direct regex pattern matching using the existing Template.pattern. Here's why this is significantly faster:

Key Optimization: Instead of creating a defaultdict and performing a full template substitution to discover variables, the optimized code directly uses the regex pattern that Template internally uses to find variable placeholders.

What Changed:

  • Eliminated the costly Template(template).safe_substitute(d) call (92% of original runtime)
  • Added direct regex matching using Template.pattern.finditer()
  • Added deduplication logic with a set to handle repeated variables
  • Extracts both named (e.g., $foo) and braced (e.g., ${foo}) variable patterns

Why It's Faster:

  1. No String Substitution: The original code performs actual template substitution which is computationally expensive, while the optimized version only parses to identify patterns
  2. Single Pass Processing: Uses finditer() to process the template once instead of the multi-step substitution process
  3. Efficient Deduplication: Uses a set for O(1) duplicate checking instead of relying on dictionary key behavior

Performance Characteristics:

  • Small templates: 40-60% faster (most common case)
  • Templates with no variables: 56-62% faster due to avoiding substitution entirely
  • Large templates with escaped dollars: Up to 91% faster since regex skips escaped patterns efficiently
  • Large templates with many variables: 27-37% faster, showing the optimization scales well

The optimization maintains identical behavior and correctness while dramatically reducing computational overhead, especially beneficial for templates with many escaped dollar signs or large-scale template processing.

Correctness verification report:

Test Status
⚙️ Existing Unit Tests 1 Passed
🌀 Generated Regression Tests 78 Passed
⏪ Replay Tests 236 Passed
🔎 Concolic Coverage Tests 🔘 None Found
📊 Tests Coverage 80.0%
⚙️ Existing Unit Tests and Runtime
Test File::Test Function Original ⏱️ Optimized ⏱️ Speedup
unit_tests/utils/test_templating_utils.py::test_get_template_variables 9.39μs 6.66μs 40.9%✅
🌀 Generated Regression Tests and Runtime
import collections
import string
from typing import List

# imports
import pytest  # used for our unit tests
from guardrails.utils.templating_utils import get_template_variables

# unit tests

# ----------------------
# Basic Test Cases
# ----------------------

def test_single_variable():
    # Test with a single variable
    template = "Hello, $name!"
    codeflash_output = sorted(get_template_variables(template)) # 8.23μs -> 5.43μs (51.6% faster)

def test_multiple_variables():
    # Test with multiple variables
    template = "Hello, $first $last!"
    result = sorted(get_template_variables(template)) # 8.76μs -> 6.03μs (45.2% faster)

def test_repeated_variables():
    # Test with repeated variables
    template = "Hello, $name! Your name is $name."
    codeflash_output = get_template_variables(template); result = codeflash_output # 8.54μs -> 5.92μs (44.2% faster)

def test_no_variables():
    # Test with no variables
    template = "Hello, world!"
    codeflash_output = get_template_variables(template) # 4.49μs -> 2.87μs (56.5% faster)

def test_adjacent_variables():
    # Test with adjacent variables
    template = "$greeting$punctuation"
    result = sorted(get_template_variables(template)) # 8.66μs -> 6.17μs (40.2% faster)

def test_braced_variables():
    # Test with braced variables
    template = "Hello, ${name}!"
    codeflash_output = get_template_variables(template) # 7.83μs -> 5.46μs (43.4% faster)

def test_mixed_braced_and_unbraced():
    # Test with both braced and unbraced variables
    template = "Hello, $name! Your ID is ${id}."
    result = sorted(get_template_variables(template)) # 8.92μs -> 6.22μs (43.4% faster)

def test_variable_with_digits():
    # Test variable names with digits
    template = "Value: $var1 and $var2"
    result = sorted(get_template_variables(template)) # 8.43μs -> 6.00μs (40.5% faster)

def test_variable_with_underscore():
    # Test variable names with underscores
    template = "Hello, $user_name!"
    codeflash_output = get_template_variables(template) # 7.79μs -> 5.26μs (47.9% faster)

def test_escaped_dollar_sign():
    # Test with escaped dollar signs
    template = "Total: $100 and $amount"
    # Only 'amount' should be detected as a variable
    codeflash_output = get_template_variables(template) # 8.81μs -> 5.77μs (52.7% faster)

# ----------------------
# Edge Test Cases
# ----------------------

def test_empty_template():
    # Test with an empty template string
    template = ""
    codeflash_output = get_template_variables(template) # 4.39μs -> 2.71μs (62.1% faster)

def test_only_dollar_sign():
    # Test with only a dollar sign
    template = "$"
    codeflash_output = get_template_variables(template) # 7.11μs -> 4.72μs (50.6% faster)

def test_dollar_sign_at_end():
    # Test with a dollar sign at the end
    template = "Total is $"
    codeflash_output = get_template_variables(template) # 7.09μs -> 4.53μs (56.3% faster)

def test_invalid_variable_name():
    # Test with invalid variable names (e.g., $1var, which is not allowed)
    template = "Value: $1var"
    # Should not detect '1var' as a variable
    codeflash_output = get_template_variables(template) # 7.10μs -> 4.76μs (49.1% faster)

def test_variable_with_braces_and_invalid_name():
    # Braced variable with invalid name
    template = "Value: ${1var}"
    # Should not detect '1var' as a variable
    codeflash_output = get_template_variables(template) # 6.99μs -> 4.67μs (49.8% faster)

def test_dollar_sign_followed_by_space():
    # Test with dollar sign followed by space
    template = "Price: $ amount"
    codeflash_output = get_template_variables(template) # 7.25μs -> 4.67μs (55.4% faster)

def test_nested_braces():
    # Test with nested braces (should not be valid)
    template = "Hello, ${user_{id}}!"
    # Only 'user_' should be detected, because '{id}' is not valid
    codeflash_output = get_template_variables(template) # 7.78μs -> 5.17μs (50.5% faster)

def test_variable_with_trailing_brace():
    # Test with a variable with trailing brace but no opening brace
    template = "Hello, $name}!"
    codeflash_output = get_template_variables(template) # 8.02μs -> 5.28μs (52.1% faster)

def test_variable_with_leading_brace():
    # Test with a variable with leading brace but no closing brace
    template = "Hello, ${name!"
    codeflash_output = get_template_variables(template) # 7.97μs -> 5.12μs (55.5% faster)

def test_variable_with_special_characters():
    # Variables with special characters are not valid
    template = "Hello, $na-me!"
    codeflash_output = get_template_variables(template) # 7.81μs -> 5.09μs (53.6% faster)

def test_variable_with_newline():
    # Variable with newline in template
    template = "Hello, $name\nYour id is $id"
    result = sorted(get_template_variables(template)) # 8.62μs -> 5.90μs (45.9% faster)

def test_variable_at_start_and_end():
    # Variable at both start and end of template
    template = "$start middle $end"
    result = sorted(get_template_variables(template)) # 8.55μs -> 6.03μs (41.8% faster)

def test_variable_with_unicode():
    # Variable name with unicode (should not be valid, only ascii letters, digits, underscore)
    template = "Hello, $имя!"
    # Only 'имя' up to first non-identifier character (should not be detected)
    codeflash_output = get_template_variables(template) # 8.30μs -> 5.08μs (63.3% faster)

# ----------------------
# Large Scale Test Cases
# ----------------------

def test_many_variables():
    # Test with a large number of variables
    template = " ".join([f"${{var{i}}}" for i in range(100)])
    result = sorted(get_template_variables(template)) # 52.9μs -> 41.4μs (27.9% faster)
    expected = [f"var{i}" for i in range(100)]

def test_large_template_with_repeated_variables():
    # Test with a large template and repeated variables
    template = " ".join(["$foo $bar $baz"] * 300)
    result = sorted(get_template_variables(template)) # 249μs -> 191μs (30.7% faster)

def test_large_adjacent_variables():
    # Test with many adjacent variables
    template = "".join([f"${{var{i}}}" for i in range(100)])
    result = sorted(get_template_variables(template)) # 50.9μs -> 41.1μs (23.8% faster)
    expected = [f"var{i}" for i in range(100)]

def test_large_template_with_no_variables():
    # Large template, no variables
    template = "Hello world! " * 500
    codeflash_output = get_template_variables(template) # 6.25μs -> 4.48μs (39.6% faster)

def test_large_template_with_escaped_dollars():
    # Large template with many escaped dollars and some variables
    template = ("$100 " * 400) + "$amount"
    codeflash_output = get_template_variables(template) # 118μs -> 63.5μs (86.6% faster)

def test_large_template_with_mixed_valid_and_invalid():
    # Large template with mix of valid and invalid variables
    valid_vars = [f"${{var{i}}}" for i in range(50)]
    invalid_vars = [f"$1var{i}" for i in range(50)]
    template = " ".join(valid_vars + invalid_vars)
    result = sorted(get_template_variables(template)) # 50.0μs -> 32.6μs (53.4% faster)
    expected = [f"var{i}" for i in range(50)]

# ----------------------
# Special/Regression Test Cases
# ----------------------

def test_variable_followed_by_non_identifier():
    # Variable followed by a non-identifier char
    template = "Hello, $name!"
    codeflash_output = get_template_variables(template) # 7.80μs -> 5.07μs (54.0% faster)

def test_variable_with_dot():
    # $foo.bar should only pick up 'foo'
    template = "User: $foo.bar"
    codeflash_output = get_template_variables(template) # 7.88μs -> 5.16μs (52.7% faster)

def test_variable_with_colon():
    # $foo:bar should only pick up 'foo'
    template = "User: $foo:bar"
    codeflash_output = get_template_variables(template) # 7.79μs -> 5.24μs (48.7% faster)

def test_variable_with_space_in_braces():
    # ${foo bar} should only pick up 'foo'
    template = "Hello, ${foo bar}!"
    codeflash_output = get_template_variables(template) # 7.78μs -> 5.05μs (54.0% faster)

def test_variable_with_double_dollar():
    # $foo should not pick up 'foo'
    template = "Amount: $foo"
    codeflash_output = get_template_variables(template) # 6.81μs -> 4.24μs (60.7% faster)

def test_variable_with_empty_braces():
    # ${} is not a valid variable
    template = "Hello, ${}!"
    codeflash_output = get_template_variables(template) # 7.42μs -> 4.68μs (58.5% faster)

def test_variable_with_only_underscore():
    # $_ is a valid variable name
    template = "Hello, $_!"
    codeflash_output = get_template_variables(template) # 7.55μs -> 5.12μs (47.4% faster)

def test_variable_with_long_name():
    # Test with a long variable name
    long_name = "a" * 100
    template = f"Hello, ${long_name}!"
    codeflash_output = get_template_variables(template) # 8.68μs -> 5.88μs (47.6% faster)

def test_variable_with_leading_digit():
    # $1foo is not valid
    template = "Hello, $1foo!"
    codeflash_output = get_template_variables(template) # 7.26μs -> 4.50μs (61.1% faster)

def test_variable_with_leading_underscore():
    # $_foo is valid
    template = "Hello, $_foo!"
    codeflash_output = get_template_variables(template) # 7.86μs -> 5.17μs (51.9% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
#------------------------------------------------
import collections
from string import Template
from typing import List

# imports
import pytest
from guardrails.utils.templating_utils import get_template_variables

# unit tests

# ----------------------------
# 1. Basic Test Cases
# ----------------------------

def test_no_variables():
    # Template with no variables should return an empty list
    codeflash_output = get_template_variables("Hello, world!") # 4.87μs -> 3.12μs (56.1% faster)

def test_single_variable():
    # Template with a single variable
    codeflash_output = get_template_variables("Hello, $name!") # 8.12μs -> 5.37μs (51.1% faster)

def test_multiple_variables():
    # Template with multiple variables
    codeflash_output = get_template_variables("Hi $first $last, welcome!"); result = codeflash_output # 8.74μs -> 6.08μs (43.8% faster)

def test_variable_with_braces():
    # Template with variable using braces
    codeflash_output = get_template_variables("Your order: ${order_id}") # 7.85μs -> 5.47μs (43.5% faster)

def test_repeated_variables():
    # Template with the same variable used multiple times
    codeflash_output = get_template_variables("$foo $foo $foo"); result = codeflash_output # 8.83μs -> 6.29μs (40.4% faster)

def test_adjacent_variables():
    # Adjacent variables without space
    codeflash_output = get_template_variables("$foo$bar"); result = codeflash_output # 8.34μs -> 5.85μs (42.6% faster)

def test_variable_with_underscore_and_digits():
    # Variable names with underscores and digits
    codeflash_output = get_template_variables("ID: $user_1, Ref: $ref2"); result = codeflash_output # 8.46μs -> 5.63μs (50.4% faster)

# ----------------------------
# 2. Edge Test Cases
# ----------------------------

def test_empty_string():
    # Empty template string should return empty list
    codeflash_output = get_template_variables("") # 4.25μs -> 2.72μs (56.3% faster)

def test_only_dollar_sign():
    # Template with only a dollar sign (not a variable)
    codeflash_output = get_template_variables("$") # 6.84μs -> 4.59μs (49.2% faster)

def test_escaped_dollar_sign():
    # Escaped dollar sign should not be treated as a variable
    codeflash_output = get_template_variables("Price: $100") # 6.82μs -> 4.41μs (54.5% faster)

def test_dollar_followed_by_non_identifier():
    # Dollar sign followed by non-identifier character is not a variable
    codeflash_output = get_template_variables("Total: $!") # 7.34μs -> 4.60μs (59.8% faster)

def test_invalid_variable_name():
    # Variable name starting with a digit is not valid in Template
    codeflash_output = get_template_variables("Value: $1var") # 7.27μs -> 4.48μs (62.3% faster)

def test_variable_with_braces_and_special_chars():
    # Braced variable with special characters inside braces (invalid)
    codeflash_output = get_template_variables("Special: ${foo-bar}") # 7.83μs -> 5.09μs (53.7% faster)

def test_variable_at_start_and_end():
    # Variable at the start and end of the template
    codeflash_output = get_template_variables("$start middle $end"); result = codeflash_output # 8.62μs -> 6.01μs (43.3% faster)

def test_variable_adjacent_to_text():
    # Variable directly adjacent to text
    codeflash_output = get_template_variables("Hello $name!") # 7.65μs -> 5.12μs (49.3% faster)

def test_malformed_variable():
    # Malformed variable (e.g., missing closing brace)
    codeflash_output = get_template_variables("Hello ${name") # 7.60μs -> 4.93μs (54.0% faster)

def test_nested_braces():
    # Nested braces are not supported; should not match as variable
    codeflash_output = get_template_variables("Hello ${user{name}}") # 7.74μs -> 5.04μs (53.6% faster)

def test_unicode_variable_name():
    # Variable names cannot contain unicode, so should not match
    codeflash_output = get_template_variables("Hello $имя") # 8.17μs -> 4.96μs (64.8% faster)

def test_variable_with_numbers_only():
    # Variable name cannot be only numbers
    codeflash_output = get_template_variables("Value: $123") # 7.26μs -> 4.69μs (54.8% faster)

def test_variable_with_mixed_case():
    # Variable names are case sensitive
    codeflash_output = get_template_variables("$Var $var"); result = codeflash_output # 8.55μs -> 5.98μs (43.0% faster)

def test_variable_with_long_name():
    # Very long variable name
    long_name = "a" * 100
    template = f"Hello ${long_name}"
    codeflash_output = get_template_variables(template) # 8.34μs -> 5.91μs (41.2% faster)

def test_template_with_comment_like_text():
    # Variable inside something that looks like a comment
    codeflash_output = get_template_variables("Hello $name # this is a comment") # 7.73μs -> 5.33μs (45.0% faster)

def test_variable_with_trailing_underscore():
    # Variable with trailing underscore
    codeflash_output = get_template_variables("Hello $name_") # 7.67μs -> 5.08μs (50.9% faster)

def test_variable_with_leading_underscore():
    # Variable with leading underscore
    codeflash_output = get_template_variables("Hello $_name") # 7.60μs -> 5.18μs (46.8% faster)

def test_variable_with_mixed_characters():
    # Variable with mixed valid characters
    codeflash_output = get_template_variables("Hello $name_123") # 7.58μs -> 5.18μs (46.4% faster)

# ----------------------------
# 3. Large Scale Test Cases
# ----------------------------

def test_many_unique_variables():
    # Template with many unique variables (up to 1000)
    variables = [f"var{i}" for i in range(1000)]
    template = " ".join([f"${v}" for v in variables])
    codeflash_output = get_template_variables(template); result = codeflash_output # 373μs -> 294μs (27.0% faster)

def test_many_repeated_variables():
    # Template with many repeated variables
    template = " ".join(["$foo"] * 1000)
    codeflash_output = get_template_variables(template); result = codeflash_output # 282μs -> 204μs (37.9% faster)

def test_large_template_with_mixed_content():
    # Large template with variables and regular text
    variables = [f"v{i}" for i in range(500)]
    text = "lorem ipsum dolor sit amet"
    template = " ".join([f"{text} ${v}" for v in variables])
    codeflash_output = get_template_variables(template); result = codeflash_output # 196μs -> 150μs (31.2% faster)

def test_large_template_with_braced_and_unbraced_vars():
    # Large template with both braced and unbraced variables
    variables = [f"var{i}" for i in range(200)]
    template = " ".join([f"${v} ${{{v}}}" for v in variables])
    codeflash_output = get_template_variables(template); result = codeflash_output # 161μs -> 125μs (29.3% faster)

def test_large_template_with_non_variable_dollars():
    # Large template with many dollar signs that are not variables
    template = " ".join(["$100"] * 1000)
    codeflash_output = get_template_variables(template); result = codeflash_output # 282μs -> 147μs (91.2% faster)

def test_large_template_with_mixed_valid_and_invalid_vars():
    # Mix valid and invalid variable patterns
    valid_vars = [f"var{i}" for i in range(300)]
    invalid_vars = [f"1var{i}" for i in range(300)]  # invalid: starts with digit
    template = " ".join([f"${v}" for v in valid_vars + invalid_vars])
    codeflash_output = get_template_variables(template); result = codeflash_output # 213μs -> 140μs (51.9% faster)

# ----------------------------
# Pytest parametrize for compactness (optional)
# ----------------------------

@pytest.mark.parametrize("template,expected", [
    ("$foo $bar $baz", {"foo", "bar", "baz"}),
    ("$foo $foo $foo", {"foo"}),
    ("Hello, world!", set()),
    ("Price: $100", set()),
    ("$foo$bar", {"foo", "bar"}),
    ("$foo $1bar", {"foo"}),
    ("$", set()),
    ("$foo $bar $foo", {"foo", "bar"}),
])
def test_various_templates(template, expected):
    # Parametrized test for various templates
    result = set(get_template_variables(template)) # 62.4μs -> 42.2μs (48.0% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
⏪ Replay Tests and Runtime
Test File::Test Function Original ⏱️ Optimized ⏱️ Speedup
test_pytest_testsunit_teststest_guard_log_py_testsintegration_teststest_guard_py_testsunit_testsvalidator__replay_test_0.py::test_guardrails_utils_templating_utils_get_template_variables 510μs 358μs 42.5%✅

To edit these changes git checkout codeflash/optimize-get_template_variables-mh2paoz2 and push.

Codeflash

The optimization replaces the expensive `safe_substitute()` operation with direct regex pattern matching using the existing `Template.pattern`. Here's why this is significantly faster:

**Key Optimization**: Instead of creating a `defaultdict` and performing a full template substitution to discover variables, the optimized code directly uses the regex pattern that `Template` internally uses to find variable placeholders.

**What Changed**:
- Eliminated the costly `Template(template).safe_substitute(d)` call (92% of original runtime)
- Added direct regex matching using `Template.pattern.finditer()`
- Added deduplication logic with a `set` to handle repeated variables
- Extracts both `named` (e.g., `$foo`) and `braced` (e.g., `${foo}`) variable patterns

**Why It's Faster**:
1. **No String Substitution**: The original code performs actual template substitution which is computationally expensive, while the optimized version only parses to identify patterns
2. **Single Pass Processing**: Uses `finditer()` to process the template once instead of the multi-step substitution process
3. **Efficient Deduplication**: Uses a `set` for O(1) duplicate checking instead of relying on dictionary key behavior

**Performance Characteristics**:
- **Small templates**: 40-60% faster (most common case)
- **Templates with no variables**: 56-62% faster due to avoiding substitution entirely
- **Large templates with escaped dollars**: Up to 91% faster since regex skips escaped patterns efficiently
- **Large templates with many variables**: 27-37% faster, showing the optimization scales well

The optimization maintains identical behavior and correctness while dramatically reducing computational overhead, especially beneficial for templates with many escaped dollar signs or large-scale template processing.
@codeflash-ai codeflash-ai bot requested a review from mashraf-222 October 23, 2025 00:45
@codeflash-ai codeflash-ai bot added the ⚡️ codeflash Optimization PR opened by Codeflash AI label Oct 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

⚡️ codeflash Optimization PR opened by Codeflash AI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants